1. Technical Field
A “Compatibility Tester” provides various techniques for extracting observable behaviors from different implementations of a system (e.g., software, operating systems, device drivers, hardware interfaces, devices, etc.) with those behaviors then being used for compatibility testing based on comparisons of various features of the extracted behaviors.
2. Background Art
When an OS, or other software or system, is upgraded or changed on a particular computing device, the question arises as to whether devices or different versions or implementations of some previously compatible connected system will remain compatible with the upgraded or changed system. For example, when upgrading a computer OS from the Windows® 7 to Windows® 8 or changing a USB controller from one standard to another (e.g., USB 2.0 to USB 3.0), there may be a question of whether all of the devices currently connected to a user's computer will remain compatible between the different versions of the OS or controller.
Various techniques for testing such compatibility issues generally deal with an evaluation of the commands and timing that hardware devices or software systems use relative to software and hardware frameworks supported by the OS. These commands include data messages or signals across various wired or wireless busses or protocols, such as PCI, USB, FireWire®, Thunderbolt™, IEEE 802.x, etc. Strategies for compatibility testing often include functional testing that involves a suite of tests to check whether the behavior of a driver or software follows a particular design document that is often referred to as a specification. Typically, such testing is performed using techniques including physically testing every device under different versions of the system, simulating a device or system via a software model, or by using hardware simulators that can be programmed to simulate various devices in particular classes of devices (e.g., input devices, output devices, etc.).
In other words, compatibility testing is aimed at verifying whether the device or Application under test interacts and functions as expected with other software or system versions. However, given the large number of existing devices and systems (numbering in the billions, with many tens of thousands of unique devices and even more unique software components), compatibility testing issues can be challenging and extremely time-consuming.
Other approaches in the domain of software testing involve model-based testing. In conventional model-based testing, the system under test is typically represented by a model describing its expected behavior at a higher abstraction level, and a set of chosen algorithms are used to generate tests from this model. In general, models are typically manually constructed based on some software specification or are created using an observation-based modeling approach. In either case, the model is typically refined and verified against a device specification or software specification to identify potential compatibility problems.
Related techniques use finite state models to validate and verify software systems. Construction of such models is often automatically generated by mining execution traces to infer statistically significant temporal properties that capture relations between non-consecutive and possibly distant events. The mined properties steer the learning of behavioral models that are then used for compatibility testing. Again, the resulting models are typically verified against a device specification or software specification to identify compatibility problems.